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TITLE OF THE INVENTION 
IMAGE SENSING APPARATUS AND METHOD, PROGRAM, AND 

STORAGE MEDIUM 

5 FIELD OF THE INVENTION 

The present invention relates to an image sensing 
apparatus having a lookup table arithmetic processing 
section, and a method, a program, and a storage medium 
therefor . 

10 

BACKGROUND OF THE INVENTION 

In recent years, to improve color reproduction of an 
image sensed by an image sensing apparatus such as a digital 
camera, more complex image processing is used to convert 

15 image data, which is obtained by A/D-converting an image 
from a CCD, into a final output image. Examples of the image 
processing are color conversion processing corresponding 
to chromatic adaptation as a human color perception 
characteristic and color conversion processing for 

20 converting a color into a memory color that is felt by humans 
as desirable. 

The present applicant has developed a processing 
method using a three-dimensional lookup table for the above 
image processing. However, in the color conversion 

25 processing corresponding to chromatic adaptation, to 

change color reproduction in accordance with a change in 
photographing light source (color temperature) , 
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three-dimensional lookup table data equal in number to 
light sources corresponding to photographing light sources 
(color temperatures) must be held, as shown in, e.g., Fig. 4 
of Japanese Patent Laid-Open No. 8-9241. However, the 
5 size of the three-dimensional lookup table data is very 
large. To hold a plurality of three-dimensional lookup 
table data, the memory capacity in the image sensing 
apparatus must be increased. To hold three-dimensional 
lookup table data equal in number to light sources is not 

10 realistic from the viewpoint of cost. There is another 
method in which three-dimensional lookup table data equal 
in number to a minimum number of light source color 
temperatures are held, and at the time of photographing, 
interpolation arithmetic processing corresponding to the 

15 light source color temperature at that time is executed 
using the three-dimensional lookup table to create 
three-dimensional lookup table data corresponding to the 
light source color temperature. However, the less the 
number of three-dimensional lookup tables becomes, the 

20 lower the color conversion accuracy becomes. In addition, 
since at least two three-dimensional lookup tables are 
still necessary, the memory capacity must also be 
increased. 

25 SUMMARY OF THE INVENTION 

The present invention has therefore been made in 
consideration of the above-described problems, and has as 
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its object to decrease the memory capacity required for a 
lookup table. 

In order to solve the above-described problems and 
achieve the object, according to the first aspect of the 
5 present invention, there is provided an image sensing 
apparatus which causes an image processing section to 
execute image processing to convert digital image data, 
which is obtained by A/D-converting an output from an image 
sensing element, into output image data, wherein the image 

10 processing section comprises a matrix arithmetic 

processing section and an N-dimensional (N is a positive 
integer) lookup table arithmetic processing section and 
causes the matrix arithmetic processing section to process 
the digital image data before the N-dimensional lookup 

15 table arithmetic processing section. 

According to the second aspect of the present 
invention, there is provided an image sensing apparatus 
which has an image sensing element and an A/D conversion 
section which A/D-converts an output from the image sensing 

20 element, causes an image processing section to execute 
image processing to convert digital image data obtained 
from the A/D conversion section into output image data, and 
records the output image data in a recording medium, wherein 
the image processing section comprises a white balance 

25 processing section, a matrix arithmetic processing 

section, and an N-dimensional lookup table arithmetic 
processing section where N is an integer not less than 3 
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and causes the matrix arithmetic processing section to 
process the digital image data before the N-dimensional 
lookup table arithmetic processing section. 

According to the third aspect of the present 
5 invention, there is provided an image sensing method which 
comprises an image processing step of sensing an image of 
an object with an image sensing element and an A/D 
conversion processing step of A/D-converting an output from 
the image sensing step, and executes image processing in 
10 an image processing step to convert digital image data 
obtained in the A/D conversion processing step into output 
image data, wherein the image processing step comprises a 
matrix arithmetic processing step and an N-dimensional (N 
is a positive integer) lookup table arithmetic processing 
15 step and processes the digital image data in the matrix 
arithmetic processing step before the N-dimensional lookup 
table arithmetic processing step. 

According to the fourth aspect of the present 
invention, there is provided a program causing a computer 
20 to execute the above image sensing method. 

According to the fifth aspect of the present 
invention, there is provided a computer-readable storage 
medium storing the above program. 

Other features and advantages of the present 
25 invention will be apparent from the following description 
taken in conjunction with the accompanying drawings, in 
which like reference characters designate the same or 
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similar parts throughout the figures thereof. 

BRIEF DESCRIPTION OF THE DRAWINGS 
Fig. 1 is a block diagram showing the schematic 
arrangement of an image sensing apparatus according to an 
embodiment of the present invention; 

Fig. 2 is a block diagram showing the arrangement of 
an image processing section; 

Fig. 3 is a diagram showing a digital signal after 
A/D conversion; and 

Fig. 4 is a diagram showing a digital image signal 
after interpolation processing. 

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT 
15 A preferred embodiment of the present invention will 

be described below in detail with reference to the 
accompanying drawings . 

Fig. 1 is a block diagram showing the schematic 
arrangement of an image sensing apparatus according to an 
20 embodiment of the present invention. 

An image sensing section 101 including a lens system, 
stop, and shutter forms the image of an object on an image 
sensing surface of a CCD 102. The object image formed on 
the CCD 102 is photoelectrically converted into an analog 
25 signal. The analog signal is sent to an A/D conversion 
section 103 and converted into a digital image signal. The 
digital image signal generated by the A/D conversion 



5 



10 



- 5 - 



section 103 is sent to an image processing section 104 and 
converted into an output image signal. The output image 
signal undergoes format conversion in a format conversion 
section 105 to obtain a JPEG format or the like. Then, the 
5 image signal is written in the internal memory of the image 
sensing apparatus or an external memory such as a compact 
flash memory (registered trademark) by an image recording 
section 106. The flow of data in the image sensing 
apparatus has been briefly described above. 

10 The image processing section 104 will be described 

in more detail. 

Fig. 2 is a block diagram showing processing sections 
included in the image processing section 104 in Fig . 1 . The 
flow of image processing in the image sensing apparatus 

15 according to this embodiment will be described below with 
reference to the block diagram shown in Fig. 2. 

The digital image signal output from the A/D 
conversion section 103 shown in Fig. 1 is sent to a white 
balance processing section 201 shown in Fig. 2. The color 

20 temperature of the light source is detected from the digital 
image signal, and a white balance coefficient that 
represents white in the image as a write signal is obtained. 
The gain of the digital image signal is adjusted using the 
obtained white balance coefficient. The digital image 

25 signal that has undergone the white balance processing is 
sent to an edge enhancement processing section 207 and 
interpolation processing section 202. In the 



interpolation processing section 202 , interpolation 
arithmetic processing is performed using pixels at 
positions R, Gl, G2, and B in the pixel layout of a single-CCD 
as shown in Fig. 3, thereby creating surface data for R, 
5 Gl, G2, and B as shown in Fig. 4. 

As is known, chromatic adaptation as a human color 
perception characteristic changes depending on the color 
temperature. For example, under evening glow or an 
incandescent lamp, the human eye cannot completely adapt 

10 and recognizes a white object as not write but a color with 
an orange tone. To cope with such chromatic adaptation of 
the human eye (to cope with a change in color reproduction 
due to a change in light source color temperature) , a matrix 
arithmetic processing section 203 determines a matrix 

15 coefficient to be used for matrix arithmetic processing on 

the basis of the light source color temperature obtained 

by the white balance processing section 201 and executes 

a matrix arithmetic operation. The matrix arithmetic 

processing is executed for each pixel using: 
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How to obtain the matrix coefficient will briefly be 
described below. 

Matrix coefficients corresponding to color 
temperatures, i.e., a matrix coefficient MTX3K for a light 
25 source color temperature of 3,000 K or less, a matrix 
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coefficient MTX4K for 4,000 K, a. matrix coefficient MTX5K 
for 5,000 K, a matrix coefficient MTX6K for 6,000 K, and 
a matrix coefficient MTX7K for 7, 000 K or more are prepared 
in advance in the matrix arithmetic processing section, 
5 Here, a matrix coefficient MTX for a light source color 
temperature of 5,300 K should be obtained. First, the 
matrix coefficients of the upper and lower light source 
color temperature levels are selected. In this case, the 
matrix coefficient MTX5K for 5,000 K and the matrix 
10 coefficient MTX6K for 6,000 K are selected. The matrix 
coefficient corresponding to the light source is obtained 
using the selected matrix coefficients in accordance with: 
MTX = (5300 - 5000) /1000 X MTX5K + 

(6000 - 5300) /1000 X MTX6K ...Equation (2) 
15 However, when the color temperature of the light source is 
3, 000 K or less, 

MTX = MTX3K ...Equation (3) 

When the color temperature of the light source is 7,000 K 
or more, 

20 MTX = MTX7K ...Equation (4) 

The digital image signal that has undergone the 
matrix arithmetic processing is sent to a gamma processing 
section 204 . The gamma processing section 204 converts the 
data using: 

25 Rg = GammaTable [Rm] ...Equation (5) 

Gg = GammaTable [Gm] ...Equation (6) 

Bg = GammaTable [Bm] ...Equation (7) 
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where GammaTable represents a one-dimensional lookup 
table. 

The digital image signal that has undergone the gamma 
processing is sent to a three-dimensional lookup table 
5 arithmetic processing section 205. 

The three-dimensional lookup table arithmetic 
processing will be described below. The three-dimensional 
lookup table arithmetic processing according to this 
embodiment converts three-dimensional data of R, G, and B 

10 from R, G, and B signals that have undergone the matrix 
processing into R, G, and B signals that take memory color 
reproduction into consideration. More specifically, R, G, 
and B signals are converted into colors that are memorized 
by humans as desirable colors. For example, a blue sky is 

15 converted from blue into blue green, grass is converted from 
yellowish green into green, and a skin is converted into 
a more pinkish flesh color. In this embodiment, to decrease 
the capacity of the three-dimensional lookup table, the 
minimum to maximum values of the R, G, and B signals are 

20 divided into nine parts to prepare 9X9X9 = 729 

three-dimensional representative lattice points. R, G, 
and B signals other than the representative lattice points 
are obtained by interpolation. Interpolation is performed 
on the basis of the equations given below. Since the image 

25 data undergoes the matrix arithmetic in advance in 

accordance with the color temperature, it is unnecessary 
to prepare a three-dimensional lookup table corresponding 
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to each color temperature. 

Let R, G, and B be the input R, G, and B signals, 
Rout(R,G,B), Gout(R,G,B), and Bout(R,G,B) be output R, G, 
and B signals, Ri, Gi, and Bi be the signals at 
representative lattice points having values which are 
smaller than and closest to the signal values, 
Rout (Ri, Gi, Bi) , Gout (Ri, Gi, Bi) , and Bout (Ri, Gi, Bi) be the 
representative lattice point output signals, and Step be 
the step width of the representative lattice points. Then, 
we have 

R = Ri + Rf 

G = Gi + Gf 

B = Bi + Bf 

Rout(R,G,B) = Rout (Ri+Rf ,Gi+Gf , Bi+Bf ) = 

(Rout (Ri,Gi,Bi) X (Step - Rf) X (Step - Gf) X 

(Step - Bf) 

+ Rout (Ri+Step,Gi,Bi) X (Rf) X (Step - Gf) X 
(Step - Bf) 

+ Rout (Ri,Gi+Step,Bi) X (Step - Rf) X (Gf) X 
(Step - Bf) 

+ Rout (Ri,Gi,Bi+Step) X (Step - Rf) X (Step - 
Gf) X (Bf) 

+ Rout (Ri+Step,Gi+Step,Bi) X (Rf) X (Gf) X (Step 
- Bf) 

+ Rout (Ri+Step,Gi,Bi+Step) X (Rf) X (Step - Gf) 
X (Bf) 

+ Rout (Ri,Gi+Step,Bi+Step) X (Step - Rf) X (Gf) 
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X (Bf) 

+ Rout (Ri+Step,Gi+Step, Bi+Step) X (Rf) X (Gf) X 
(Bf))/(Step X Step X Step) ...Equation (8) 

Gout(R,G,B) = Gout (Ri+Rf ,Gi+Gf , Bi+Bf ) = 
(Gout (Ri,Gi,Bi) X (Step - Rf) X (Step - Gf) X 
(Step - Bf) 

+ Gout (Ri+Step,Gi,Bi) X (Rf) X (Step - Gf) X 
(Step - Bf) 

+ Gout (Ri,Gi+Step,Bi) X (Step - Rf) X (Gf) X 
(Step - Bf) 

+ Gout (Ri,Gi, Bi+Step) X (Step - Rf) X (Step - 
Gf) X (Bf) 

+ Gout (Ri+Step,Gi+Step,Bi) X (Rf) X (Gf) X (Step 
- Bf) 

+ Gout (Ri+Step,Gi,Bi+Step) X (Rf) X (Step - Gf) 
X (Bf) 

+ Gout (Ri,Gi+Step, Bi+Step) X (Step - Rf) X (Gf) 
X (Bf) 

+ Gout (Ri+Step,Gi+Step, Bi+Step) X (Rf) X (Gf) X 
(Bf))/(Step X Step X Step) ...Equation (9) 

Bout(R,G,B) = Bout (Ri+Rf, Gi+Gf, Bi+Bf ) = 
(Bout (Ri,Gi,Bi) X (Step - Rf) X (Step - Gf) X 
(Step - Bf) 

+ Bout (Ri+Step,Gi,Bi) X (Rf) X (Step - Gf) X 
(Step - Bf) 

+ Bout (Ri,Gi+Step,Bi) X (Step - Rf) X (Gf) X 
(Step - Bf) 
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+ Bout (Ri, Gi,Bi+Step) X (Step - Rf) X (Step - 
Gf) X (Bf) 

+ Bout (Ri+Step,Gi+Step,Bi) X (Rf) X (Gf) X (Step 
- Bf ) 

+ Bout(Ri+Step,Gi,Bi+Step) X (Rf) X (Step - Gf) 
X (Bf) 

+ Bout (Ri,Gi+Step, Bi+Step) X (Step - Rf) X (Gf) 
X (Bf) 

+ Bout (Ri+Step,Gi+Step, Bi+Step) X (Rf) X (Gf) X 
(Bf))/(Step X Step X Step) ...Equation (10) 

Here, the input signals Rf , Gf , and Bf correspond to 
gamma-processed digital image signals Rg, Gg, and Bg, and 
the output signals Rout, Bout, and Gout correspond to Rt, 
Bt, and Gt, respectively. 

Using the above arithmetic processing, the input R, 
G, and B signals (Rg, Gg, and Bg) are converted into output 
R, G, and B signals (Rt, Gt, and Bt) . 

The digital image signal that has undergone the 
three-dimensional lookup table arithmetic processing is 
sent to an edge synthesis processing section 206. 

In the edge enhancement processing section 207, edges 
are detected from the digital image signal which is 
white-balance-processed and sent from the white balance 
processing section 201, and only an edge signal is 
extracted. The extracted edge signal is amplified by gain 
amplification and sent to the edge synthesis processing 
section 206. The edge synthesis processing section 206 
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adds the edge signal to the Rt, Gt, and Bt signals sent from 
the three-dimensional lookup table arithmetic processing 
section 205. 

In this embodiment, the sets of matrix coefficients 
5 are prepared in advance in the matrix arithmetic processing 
section every 1,000 K from 3,000 K to 7,000 K. However, 
the color temperatures and the number of matrix 
coefficients prepared in advance are not limited to these. 
The number of sets of matrix coefficients may be increased 

10 to more accurately cope with a change in color reproduction 
due to a change in light source color temperature. Even 
when the number of sets of matrix coefficients is increased, 
it does not always increase the memory capacity because the 
data amount of one set of matrix coefficients is small, as 

15 compared to the case wherein the number of lookup tables 
is increased by one. 

The table in the three-dimensional lookup table 
arithmetic processing section 205 uses 9X9X9 lattice 
points. However, the present invention is not limited to 

20 this. The number of lattice points may be changed in 

accordance with the memory capacity of the image sensing 
apparatus and the required color conversion accuracy. 
Interpolation arithmetic processing from near lattice 
points is not limited to equations (8) , (9) , and (10) . Any 

25 other interpolation arithmetic processing such as 
tetrahedral interpolation may be used. 

In this embodiment, gamma processing is executed 
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before three-dimensional lookup table arithmetic 
processing. However, the order of processing operations 
is not limited to this, and either processing can be 
executed first on the basis of the relationship between the 
5 arithmetic accuracy and the buffer memory of the image 
sensing apparatus. For example, to decrease the buffer 
memory capacity, gamma processing is executed first to 
decrease the number of bits of the digital image signal and 
then, the three-dimensional lookup table arithmetic 

10 processing is executed. In this case, the number of bits 
of the lattice point data of the three-dimensional lookup 
table can also be decreased. Hence, the capacity of the 
three-dimensional lookup table data can be decreased. 
Conversely, when the three-dimensional lookup table 

15 arithmetic processing is executed before the number of bits 
is decreased, the conversion arithmetic processing can be 
executed without decreasing the arithmetic accuracy. 

As described above, according to the above 
embodiment, digital image data (R, G, and B signals) 

20 obtained by A/D-converting image data from the CCD 102 is 
processed by the matrix arithmetic processing section 203 
before the processing by the three-dimensional lookup table 
arithmetic processing section 205. For this reason, it is 
unnecessary to prepare a three-dimensional lookup table for 

25 each color temperature. The memory color conversion can 
be executed using only one three-dimensional lookup table. 

In the above embodiment, a three-dimensional lookup 
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table is used. However, the present invention can be 
applied not only to a three-dimensional lookup table but 
also an arbitrary N-dimensional (N is a positive integer) 
lookup table such as a one-dimensional lookup table 
5 prepared for each of the R, G, and B colors, a 

two-dimensional lookup table, or a four-dimensional lookup 
table for C, M, Y, and G. 
[Other Embodiment] 

The present invention may be applied to a system 

10 constituted by a plurality of devices (e.g., a host 

computer, an interface device, a reader, a printer, and the 
like) or an apparatus comprising a single device (e.g., a 
copying machine, a facsimile apparatus, or the like) . 

The object of the present invention is achieved even 

15 by supplying a storage medium (or recording medium) which 
stores the program codes of software that implements the 
functions of the above-described embodiment to the system 
or apparatus and causing the computer (or CPU or MPU) of 
the system or apparatus to read out and execute the program 

20 codes stored in the storage medium. In this case, the 
program codes read out from the storage medium implement 
the functions of the above-described embodiment by 
themselves, and the storage medium which stores the program 
codes constitutes the present invention. The functions of 

25 the above-described embodiment are implemented not only 
when the readout program codes are executed by the computer 
but also when the operating system (OS) running on the 
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computer performs part or all of actual processing on the 
basis of the instructions of the program codes. 

The functions of the above-described embodiment are 
also implemented when the program codes read out from the 
5 storage medium are written in the memory of a function 
expansion card inserted into the computer or a function 
expansion unit connected to the computer, and the CPU of 
the function expansion card or function expansion unit 
performs part or all of actual processing on the basis of 
10 the instructions of the program codes. 

As described above, according to the above 
embodiments, the memory capacity required for the lookup 
table can be decreased. 

As many apparently widely different embodiments of 
15 the present invention can be made without departing from 
the spirit and scope thereof, it is to be understood that 
the invention is not limited to the specific embodiments 
thereof except as defined in the appended claims. 
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